Sensitivity lens for assessing uncertainty in image visualizations of data sets, related methods and computer products

ABSTRACT

Systems for rendering images from a multi-dimensional data set include a sensitivity lens configured to electronically render a sequence of images showing features in a region of interest to interrogate and/or explore potential uncertainties in the rendered visualizations of images.

FIELD OF THE INVENTION

The present invention relates to medical visualizations of image dataand may be particularly suitable for viewing important diagnosticfeatures in medical images.

RESERVATION OF COPYRIGHT

A portion of the disclosure of this patent document contains material towhich a claim of copyright protection is made. The copyright owner hasno objection to the facsimile or reproduction by anyone of the patentdocument or the patent disclosure, as it appears in the Patent andTrademark Office patent file or records, but reserves all other rightswhatsoever.

BACKGROUND OF THE INVENTION

Uncertainty in visualization is an important area within medical imagingwhere adequate technical solutions have not been proposed. Beforedigital images, in the era of film-based radiology, physicians wererestricted to the single visualization represented by the film. However,there now are a large number of ways to present a medical image, i.e.,ways to visualize the digital data. Having the images in digital formatmeans that the user can adjust many parameters in order to visualizedifferent aspects of the data. This means that there can be otherrelevant alternative visualizations that a radiologist or otherclinician should consider before making a diagnosis based on a singlevisualization. In other words, these alternatives can constitute anuncertainty in the visualization that should be explored, as whenuncertainty is not considered, the diagnostic assessment may beincorrect.

SUMMARY OF EMBODIMENTS OF THE INVENTION

Embodiments of the present invention are directed to methods, systemsand computer program products providing a sensitivity lens that canexplore and/or expose, uncertainty in visualizations, which can beparticularly useful in diagnostic medical visualizations.

Embodiments of the invention are particularly suitable for PACS (PictureArchiving and Communication System), however, embodiments of theinvention are more generally applicable, and should not be interpretedto be limited to PACS.

Some embodiments are directed to visualization systems for renderingimages from a multi-dimensional data set. The systems include avisualization system with a display and a graphic user input configuredto apply a sensitivity lens to a region of interest in an image renderedfrom a multi-dimensional image data set. The sensitivity lens causes thevisualization system to automatically electronically render differentvisualizations of the region of interest to interrogate uncertainty inthe rendered image associated with the visualization of at least onefeature in the region of interest.

The uncertainty interrogation may be used to illustrate a visual impactof at least one of the following: (a) different viewing parameters or(b) different processing parameters, to thereby allow a clinician toconsider uncertainty in the rendered image associated with thevisualization of at least one feature in the region of interest. Thedifferent visualizations can be displayed as an animated sequence.

Other embodiments are directed to graphic user interface tools. Thetools include at least one user-selectable sensitivity lens thatcooperates with a display to allow a user to select a region of interestin a rendered diagnostic medical image to automatically electronicallygenerate different visualization versions of the region of interestassociated with uncertainty in visualization of the rendered image.

Still other embodiments are directed to methods of assessing uncertaintyin rendering images of features in medical visualization systems. Themethods include: (a) accepting user input to apply at least onesensitivity lens to a region of interest in a visualization of arendered image; and (b) automatically displaying differentvisualizations of the region of interest defined by the sensitivity lensto assess uncertainty in the visualization of the rendered image.

Additional embodiments are directed to signal processor circuits thatinclude a rendering module for rendering images from respective patientmultidimensional imaging data sets obtained from one or severaldifferent imaging modalities. The circuit is configured to communicatewith a graphical user interface associated with a client workstation toaccept user input to apply at least one sensitivity lens to a region ofinterest in a visualized medical image rendered from a multi-dimensionalimage data set whereby a sequence of different visualizations of theregion of interest in the sensitivity lens is displayed in quicksuccession, typically between about 1-50 frames per second.

Some embodiments are directed to computer program products for providinga visualization uncertainty assessment tool for visualizations ofrendered diagnostic medical images. The computer program productsinclude a computer readable storage medium having computer readableprogram code embodied in the medium. The computer-readable program codeincluding computer readable program code configured to generate asequence of different visualizations of a region of interest in an imagerendered from a multi-dimensional image data set. The sequence ofdifferent visualizations generated can visually explore uncertainty inthe visualization of features in the region of interest.

The sequence of different visualizations can be used to illustrate avisual impact of different viewing and/or processing parameters.

Still other embodiments are directed to visualization systems forrendering diagnostic medical images from a multi-dimensional data set.The systems include a visualization system configured to apply atransfer function with an explicit probabilistic model to classify imagedata based on a likelihood of material type and map an intensity valueto an array of pure material colors to render a medical image from amulti-dimensional image data set.

The visualization systems may be configured to generate an animation offrames of different visualizations of a region of interest and run theframes in at least one of material sync animation, random mode animationand grouped random mode animation.

Yet other embodiments are directed to visualization systems forrendering diagnostic medical images from a multi-dimensional data set.The systems include a visualization system configured to substantiallyautomatically electronically generate and display an animation ofdifferent visualizations showing potential visualization uncertainty inat least part of a diagnostic medical image.

It is noted that any of the features claimed with respect to one type ofclaim, such as a system, apparatus, circuit or computer program, may beclaimed or carried out as any of the other types of claimed operationsor features.

Further features, advantages and details of the present invention willbe appreciated by those of ordinary skill in the art from a reading ofthe figures and the detailed description of the preferred embodimentsthat follow, such description being merely illustrative of the presentinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a schematic diagram of an electronic visualization system thatcan be used to render and display images

FIG. 2 is a schematic illustration of an imaging visualization systemsuch as a PACS according to embodiments of the present invention.

FIG. 3 is a screen shot of a conventional rendering of an MR renalangiography, superimposed with an arrow that points to a suspectedabnormality.

FIGS. 4A-4C are screen shots of the image shown in FIG. 3, modified witha sensitivity lens applied to the suspected abnormality region accordingto embodiments of the invention. The sensitivity lens reveals that thereis no stenosis, contrary to what could be suspected based on thevisualization image in FIG. 3. FIG. 4A-C shows uncertainty in the regionof the image using a material sync mode.

FIG. 5A is a screen shot of CT angiography visualized using DVR and apre-defined Transfer Function (TF) to assess a possible stenosis in thevessel pointed out by the arrow.

FIG. 5B is a screen shot of the same feature rendered using alternateTransfer Function settings from that used in FIG. 5A and shows thatthere is no stenosis.

FIG. 6A is a block diagram of a data processing system with asensitivity lens module according to embodiments of the presentinvention.

FIGS. 6B and 6C are schematic illustrations of a sensitivity lens toolaccording to embodiments of the present invention.

FIG. 7 is a schematic illustration of examples of different sources ofuncertainty in visualization systems according to embodiments of thepresent invention.

FIG. 8 is a graph of a window of on-screen color (black to white) versusintensity.

FIGS. 9A and 9B are graphs of transfer function models with materialspecific components. FIG. 9A is a graph of opacity versus intensity of aconventional model where each material corresponds to a varying opacityacross the data value range (intensity). FIG. 9B is a graph oflikelihood (probability) versus intensity of a probabilistic model,where each material corresponds to a varying material likelihood acrossthe data value range (intensity) and both a static color and a staticopacity according to embodiments of the present invention.

FIG. 10A is a graph of likelihood of material (color) versus intensityassociated with an explicitly probabilistic transfer function modelaccording to embodiments of the present invention.

FIG. 10B is a graph of likelihood of material (color) versus intensityassociated with an explicitly probabilistic transfer function model usedto generate the animation frames images in FIGS. 4A-C according toembodiments of the present invention.

FIGS. 11A and 11B are screen shots of MR (Magnetic Resonance) renalimages with FIG. 11A corresponding to a traditional rendering and FIG.11B desaturated using a probabilistic model according to embodiments ofthe present invention.

FIG. 12 is a schematic of a traditional DVR (direct volume rendering) TFmapping versus a probabilistic animation model according to embodimentsof the invention.

FIG. 13 is a schematic illustration of a fragment shader implementationof probabilistic uncertainty animation according to embodiments of thepresent invention.

FIGS. 14 a-14 k are examples of probabilistic animation according toembodiments of the present invention. FIG. 14 a is a traditionalrendering. FIG. 14 b is a traditional TF (opacity versus intensity) usedfor the rendering in FIG. 14 a. FIG. 14 c is an explicitly probabilistictransfer function used for animation. FIGS. 14 d-14 f are frames ofprobabilistic animation in random mode. FIG. 14 g is a schematicillustration of intensity to color mapping in random mode. FIGS. 14 h-14j are frames from probabilistic animation in material sync mode. FIG. 14k is a schematic illustration of intensity to color mapping in colormapping in material sync mode.

FIG. 15 is a schematic illustration of three different probabilisticanimation modes, random mode, grouped random mode and material sync modeaccording to embodiments of the present invention.

FIG. 16 is a schematic illustration of a general case of probabilisticanimation according to embodiments of the present invention.

FIGS. 17A-17D illustrate a simulated stenosis study. FIG. 17A is anexample of a rendered image frame in material sync mode of a vesselmodel. FIGS. 17B and 17C are cut-outs (red frame of full image) of thevessel at two other time steps. FIG. 17D is a schematic of the vesselmodel.

FIGS. 18A-18C are frames of uncertainty animation in material sync modefor a MR renal angiography according to embodiments of the presentinvention.

FIG. 18D is a screen shot from an application with a GUI to select aregion of interest for uncertainty assessment to run the uncertaintyanimation as shown in FIGS. 18A-18C according to embodiments of thepresent invention.

FIG. 18E is a probabilistic transfer function (likelihood versusintensity) used to generate the animation frames shown in FIGS. 18A-18Caccording to embodiments of the present invention.

FIGS. 19A-19F illustrate a CT (Computed Tomography) examination of athyroid tumor. FIG. 19A is a conventional rendering. FIGS. 19B-19D areframes of uncertainty animation in material sync mode according toembodiments of the present invention. FIG. 19E is a traditional TF usedto generate the image in FIG. 19A. FIG. 19F is a probabilistic transferfunction (likelihood versus intensity) used to generate the animationframes shown in FIGS. 19C-19D.

FIGS. 20A-20D illustrate an MR brain examination. FIG. 20A illustrates atissue complex in the right brain hemisphere having a cyst (a), ahemorrhage (b), and a ventricle (c), all three in the yellow-orange spanin an actual color image. The complex is surrounded by an edema (green)(in an actual color image). The diagnostic task is to determine theborders between the cyst and the hemorrhage. FIG. 20A is a traditionalrendering. FIG. 20B is a probabilistic TF. FIGS. 20C and 20D are twodifferent zoom-ins of uncertainty animation frames in random mode.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The present invention now is described more fully hereinafter withreference to the accompanying drawings, in which embodiments of theinvention are shown. This invention may, however, be embodied in manydifferent forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art.

Like numbers refer to like elements throughout. In the figures, thethickness of certain lines, layers, components, elements or features maybe exaggerated for clarity. Broken lines illustrate optional features oroperations unless specified otherwise.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein, the term “and/or”includes any and all combinations of one or more of the associatedlisted items. As used herein, phrases such as “between X and Y” and“between about X and Y” should be interpreted to include X and Y. Asused herein, phrases such as “between about X and Y” mean “between aboutX and about Y.” As used herein, phrases such as “from about X to Y” mean“from about X to about Y.”

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the specification andrelevant art and should not be interpreted in an idealized or overlyformal sense unless expressly so defined herein. Well-known functions orconstructions may not be described in detail for brevity and/or clarity.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions, layersand/or sections, these elements, components, regions, layers and/orsections should not be limited by these terms. These terms are only usedto distinguish one element, component, region, layer or section fromanother region, layer or section. Thus, a first element, component,region, layer or section discussed below could be termed a secondelement, component, region, layer or section without departing from theteachings of the present invention. The sequence of operations (orsteps) is not limited to the order presented in the claims or figuresunless specifically indicated otherwise.

The term “Direct Volume Rendering” or DVR is well known to those ofskill in the art. DVR comprises electronically rendering a medical imagedirectly from data sets to thereby display visualizations of targetregions of the body, which can include color as well as internalstructures, using multi-dimensional 3D or 4D or more dimensional data.In contrast to conventional iso-surface graphic constructs, DVR does notrequire the use of intermediate graphic constructs (such as polygons ortriangles) to represent objects, surfaces and/or boundaries. However,DVR can use mathematical models to classify certain structures and canuse graphic constructs. The term “Transfer Function” or “TF” is wellknown to those of skill in the art. A TF acts as a classification tooland maps sample values in a data set to visual appearance parameters,such as color and opacity.

Also, although embodiments of the present invention are directed to DVRof medical images, other image generation techniques and othermulti-dimensional 3-D or 4-D image data may also be used. That is, the3-D images with respective visual characteristics or features may begenerated differently when using non-DVR techniques.

Two-dimensional (2-D), three-dimensional (3-D) and four-dimensional(4-D) visualization products for providing medical images can employrendering techniques to create images from stored electronic data files.The data input used to create the image renderings can be a stack ofimage slices from a desired imaging modality, for example, a ComputedTomography (CT) or Magnetic Resonance (MR) modality and/or fromcombinations thereof. The visualization can convert the image data intoan image volume to create renderings that can be displayed on aworkstation display. Image visualizations using the multi-dimensionalimage data can be carried out using any suitable system such as, forexample, PACS. PACS is a system that receives images from the imagingmodalities, stores the data in archives, and distributes the data toradiologists and clinicians for viewing.

The term “automatically” means that the operation can be substantially,and typically entirely, carried out without human or manual input, andis typically programmatically directed or carried out. The term“electronically” includes both wireless and wired connections betweencomponents. The term “image quality” in a medical image context refersto diagnostically relevant content in the rendered image. Full or highquality means that important anatomical and/or functional features areshown at as high precision as the resolution of the original data setpermits. Low quality means that features are shown with less precisionor focus. The term “sensitivity lens” describes a graphic user interface(GUI), typically a frame or window, that is in communication with animage rendering and/or display circuit that can allow a clinician toselect and electronically explore uncertainties in a region of interestin the visualization(s) of the image on a display. The terms “imageuncertainty” or “visualization uncertainty” and the like refer topossible differences in appearance of the medical image due toalternative ways of visualizing and/or processing the image data.

The term “clinician” means physician, radiologist, physicist, or othermedical personnel desiring to review medical data of a patient. The term“tissue” means blood, cells, bone and the like. “Distinct or differenttissue” or “distinct or different material” means tissue or materialwith dissimilar density or other structural or physicallycharacteristic. For example, in medical images, different or distincttissue or material can refer to tissue having biophysicalcharacteristics different from other (local) tissue. Thus, a bloodvessel and spongy bone may have overlapping intensity but are distincttissue. In another example, a contrast agent can make tissue have adifferent density or appearance from blood or other tissue.

Visualization means to present medical images to a user/clinician forviewing. The visualization can be in a flat 2-D and/or in 2-D whatappears to be 3-D images on a display, data representing features withdifferent visual characteristics such as with differing intensity,opacity, color, texture and the like. The images as presented by thevisualization do not have to be the same as the original construct(i.e., they do not have to be the same 2-D slices from the imagingmodality). Two common visualization techniques (apart from viewingoriginal slices) are Multiplanar Reconstruction (MPR), which shows anarbitrary oblique slice through the anatomy and Maximum IntensityProjection (MIP) where a slab is visualized by displaying the maximumvalue “seen” from each image pixel. For MPR, there are a number ofvariants, the slice can be thin or constructed by averaging a thickerslab, etc.

The term “animation” refers to a sequence or series of images shown insuccession, typically in relatively quick succession, such as in about1-50 frames per second. The term “frame” refers to a singlevisualization or static image. The term “animation frame” refers to oneimage frame of the different images in the sequence of images. The term“probabilistic animation” refers to an animation technique where theanimation frames are created based on a classification of features in adata set, where the classification is based on statistic probability.The term “probabilistic transfer” function refers to a transfer functionwhere the inherent classification is defined explicitly in terms ofprobabilities. The term “parameter perturbation” refers to a controlledelectronic alteration and/or variation of a processing and/or renderingparameter typically resulting in a different visualization of a commonregion of interest in an image.

The term “similar examination type” refers to corresponding anatomicalregions or features in images having diagnostic or clinical interest indifferent data sets corresponding to different patients (or the samepatient at a different time). For example, but not limited to, acoronary artery, organs, such as the liver, heart, kidneys, lungs,brain, and the like.

In the medical image case, important diagnostic features usuallycorresponds to a particular tissue, such as bone, blood vessels, blood,brain tissue (white or gray matter), skin, cartilage, tendon, ligament,etc.

A data set can be defined as a number of grid points in G dimensions,where there is V number of values in each grid point. The term“multi-dimensional” refers to both components, grid G and variates V, ofthe data sets. For data sets having a V≧1, the data set is referred toas multi-variate. Examples: a normal medical data set has G=3 and V=1, anormal time-dependent volume has G=4 and V=1, a volume describing flowwill have G=3 and V=3 (three values, since the velocity is a 3D vector).The data sets of the instant invention for medical images will typicallyhave G and V values of: G≦4 and V≦6.

In the description that follows, a client-server setup is illustrated,but the data retrieval interfaces contemplated by the instant inventionmay be implemented within one computer as well. The term “client” willbe used both to denote a computer and the software (application) runningon the computer. Additional computers can be used including more thanone server and/or more than one client for a workstation. For example,the server can be more than one server with different functions carriedout by or between different servers, such as the patient data short orlong-term storage can be on one or more separate servers.

Turning now to FIG. 1, an exemplary visualization system 10 isillustrated. As known to those of skill in the art, the system 10 caninclude at least one server 20 s with an image import module 15, patientdata storage 20, a data fetch module 21, a client (and/or workstation)30 and a rendering system 25. The visualization system 10 can be incommunication with at least one imaging modality 11 that electronicallyobtains respective volume data sets of patients and can electronicallytransfer the data sets to the electronic storage 20. The imagingmodality 11 can be any desirable modality such as, but not limited to,NMR, MRI, X-ray of any type, including, for example, CT (computedtomography) and fluoroscopy, ultrasound, and the like. The visualizationsystem 10 may also operate to render images using data sets from morethan one of these modalities. That is, the visualization system 10 maybe configured to render images irrespective of the imaging modality datatype (i.e., a common system may render images for both CT and MRI volumeimage data). In some embodiments, the system 10 may optionally combineimage data sets generated from different imaging modalities 11 togenerate a combination image for a patient.

The rendering system 25 can be in communication with a physicianworkstation 30 to allow user input (typically graphical user input(“GUI”)) and interactive collaboration of image rendering to give thephysician alternate image views of the desired features in generally,typically substantially, real time. The rendering system 25 can beconfigured to zoom, rotate, and otherwise translate to give thephysician visualization of the patient data in one or more views, suchas section, front, back, top, bottom, and perspective views. Therendering system 25 may be wholly or partially incorporated into thephysician workstation 30, or can be a remote or local module (or acombination remote and local module) component or circuit that cancommunicate with a plurality of physician workstations (not shown). Thevisualization system can employ a computer network and may beparticularly suitable for clinical data exchange/transmission over anintranet. A respective workstation 30 can include at least one display31 (and may employ two or more adjacent displays). The workstation 30and/or rendering system 25 form part of an image processor system thatincludes a digital signal processor and other circuit components thatallow for collaborative interactive user input using the display at theworkstation 30. Thus, in operation, the image processor system rendersthe visualization of the medical image using the medical image volumedata, typically on at least one display at the physician workstation 30.

As shown in FIG. 2, each respective workstation 30 can be described as aclient 30 (shown as 30 a, 30 b, 30 c, . . . 30 e) that typicallyincludes or communicates with at least one display 31 and communicateswith at least one (hub or remote) server 20 s that stores the patientdata sets or is in communication with the stored patient electronic datafiles 20. Additional numbers of clients 30 may be in communication withthe server 20 s and more than one server 20 s may be used to storepatient data. A data retrieval interface 50 can be used to communicatewith the clients 30 a-30 e and the stored data sets on and/or accessiblevia server 20 s. Some of the clients, shown as clients 30 a, 30 b, 30 ccan be local (within a common clinic or facility) and can access thedata sets via a relatively broadband high speed connection using, forexample, a LAN, while others, shown as clients 30 d, 30 e, designated bythe broken line, may be remote and/or may have lesser bandwidth and/orspeed, and for example, may access the data sets via a WAN and/or theInternet. Firewalls may be provided as appropriate for security.

For ease of discussion, the data retrieval interface 50 is shown as astand-alone module or circuit. However, the interface 50 can be disposedpartially on each client 30, partially or wholly on the server 20 s, ormay be configured as a discrete data retrieval interface server 50 s(not shown). The clients 30, server 20 s and/or interface 50 can eachinclude a digital signal processor, circuit and/or module that can carryout aspects of the present invention. All or selected ones of theclients 30 a-30 e (FIG. 2) can be online at the same time and may eachrepeatedly communicate with the data retrieval interface 50 to requestvolume image data.

Conventionally, radiologists or other clinicians have manually exploredvisualization parameters, such as, for example, greyscale windowing,corresponding to contrast and brightness levels of the image. Differentsettings can give dramatically different appearance that, in turn, canlead to different diagnostic assessments. Radiologists typically applymanual adjustments to explore the possible alternatives. In the case of3D visualizations achieved with Direct Volume Rendering (DVR), thecorresponding component is the Transfer Function (TF), whose alternativeparameter settings also constitute an uncertainty that would benefitfrom exploration. One example is shown in FIGS. 5A and 5B. FIG. 5A is arendering using a pre-defined Transfer Function. This visualizationindicates a probable stenosis as noted by the arrow. However, usingother Transfer Function settings, as shown in FIG. 5B, the visualizationshows that there is no stenosis. Thus, when uncertainty is notconsidered in the rendered visualization, diagnostic assessment can beincorrect, potentially causing an unnecessary surgical procedure.

FIG. 3 illustrates a conventional rendering of an MR renal angiography.A potential abnormality (stenosis) appears in the image as noted by thearrow. As shown in FIGS. 4A-4C, a sensitivity lens 100 placed over thesuspected abnormality can electronically and substantially automaticallygenerate at least one alternate visualization (typically a plurality inan animated sequence) of a subset of the displayed image. The alternatevisualization(s) can allow a clinician to evaluate uncertainty invisualizations of a common feature in the different diagnosticvisualization images using the same patient image data. The sensitivitylens 100 can be manually or automatically placed over a portion of animage to programmatically generate and sequentially display alternatevisualizations of the underlying portion of the image based on knownpotential sources of uncertainty in an (medical) imaging pipeline toprovide alternate visualizations of a patient's image data (e.g.,regions or features of interest). The sensitivity lens 100 can generatevisualizations based on a probabilistic animation of differentvisualizations based on different uncertainty sources. The remainingportion(s) of the image (outside the sensitivity lens) can be unchangedfrom the original visualization. The window or frame of the sensitivitylens can “zoom” or enlarge an anatomical feature(s) of interest in thewindow or frame for ease, of viewing the alternate visualizations forclarity in uncertainty review.

As will be appreciated by one of skill in the art, embodiments of theinvention may be embodied as a method, system, data processing system,or computer program product. Accordingly, the present invention may takethe form of an entirely software embodiment or an embodiment combiningsoftware and hardware aspects, all generally referred to herein as a“circuit” or “module.” Furthermore, the present invention may take theform of a computer program product on a computer-usable storage mediumhaving computer-usable program code embodied in the medium. Any suitablecomputer readable medium may be utilized including hard disks, CD-ROMs,optical storage devices, a transmission media such as those supportingthe Internet or an intranet, or magnetic or other electronic storagedevices.

Computer program code for carrying out operations of the presentinvention may be written in an object oriented programming language suchas Java, Smalltalk or C++. However, the computer program code forcarrying out operations of the present invention may also be written inconventional procedural programming languages, such as the “C”programming language or in a visually oriented programming environment,such as VisualBasic.

Certain of the program code may execute entirely on one or more of theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer. In the latter scenario, theremote computer may be connected to the user's computer through a localarea network (LAN) or a wide area network (WAN), or the connection maybe made to an external computer (for example, through the Internet usingan Internet Service Provider). In some embodiments, some program codemay execute on local computers and some program code may execute on oneor more local and/or remote server. The communication can be done inreal time or near real time or off-line using a volume data set providedfrom the imaging modality.

The invention is described in part with reference to flowchartillustrations and/or block diagrams of methods, systems, computerprogram products and data and/or system architecture structuresaccording to embodiments of the invention. It will be understood thateach block of the illustrations, and/or combinations of blocks, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general-purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the block or blocks.

These computer program instructions may also be stored in acomputer-readable memory or storage that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory or storage produce an article of manufacture includinginstruction means which implement the function/act specified in theblock or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing the functions/acts specified inthe block or blocks.

As illustrated in FIG. 6A, embodiments of the invention may beconfigured as a data processing system 216, which can be used to carryout or direct operations of the rendering, and can include a processorcircuit 200, a memory 236 and input/output circuits 246. The dataprocessing system may be incorporated in, for example, one or more of apersonal computer, workstation 30, server, router or the like. Thesystem 116 can reside on one machine or between a plurality of machines.The processor 200 communicates with the memory 236 via an address/databus 248 and communicates with the input/output circuits 246 via anaddress/data bus 249. The input/output circuits 246 can be used totransfer information between the memory (memory and/or storage media)236 and another computer system or a network using, for example, anInternet protocol (IP) connection. These components may be conventionalcomponents such as those used in many conventional data processingsystems, which may be configured to operate as described herein.

In particular, the processor 200 can be commercially available or custommicroprocessor, microcontroller, digital signal processor or the like.The memory 236 may include any memory devices and/or storage mediacontaining the software and data used to implement the functionalitycircuits or modules used in accordance with embodiments of the presentinvention. The memory 236 can include, but is not limited to, thefollowing types of devices: ROM, PROM, EPROM, EEPROM, flash memory,SRAM, DRAM and magnetic disk. In some embodiments of the presentinvention, the memory 236 may be a content addressable memory (CAM).

As further illustrated in FIG. 6A, the memory (and/or storage media) 236may include several categories of software and data used in the dataprocessing system: an operating system 252; application programs 154;input/output device drivers 258; and data 256. As will be appreciated bythose of skill in the art, the operating system 252 may be any operatingsystem suitable for use with a data processing system, such as IBM®,OS/2®, AIX® or zOS® operating systems or Microsoft® Windows®95,Windows98, Windows2000, Windows Vista, or WindowsXP operating systemsUnix or Linux™. IBM, OS/2, AIX and zOS are trademarks of InternationalBusiness Machines Corporation in the United States, other countries, orboth while Linux is a trademark of Linus Torvalds in the United States,other countries, or both. Microsoft and Windows are trademarks ofMicrosoft Corporation in the United States, other countries, or both.The input/output device drivers 258 typically include software routinesaccessed through the operating system 252 by the application programs154 to communicate with devices such as the input/output circuits 246and certain memory 236 components. The application programs 154 areillustrative of the programs that implement the various features of thecircuits and modules according to some embodiments of the presentinvention. Finally, the data 256 represents the static and dynamic dataused by the application programs 154 the operating system 252 theinput/output device drivers 258 and other software programs that mayreside in the memory 236.

The data 256 may include (archived or stored) multi-dimensional patientdigital image data sets 226 that provides stacks of image datacorrelated to respective patients. As further illustrated in FIG. 6A,according to some embodiments of the present invention applicationprograms 154 include a Sensitivity Lens Module 225. The applicationprograms may also optionally include a DVR Module (220) and a datainterface module can be decoupled or isolated from the DVR module. Theapplication program 154 may be located in a local server (or processor)and/or database or a remote server (or processor) and/or database, orcombinations of local and remote databases and/or servers.

While the present invention is illustrated with reference to theapplication programs 154 in FIG. 6A, as will be appreciated by those ofskill in the art, other configurations fall within the scope of thepresent invention. For example, rather than being application programs154 these circuits and modules may also be incorporated into theoperating system 252 or other such logical division of the dataprocessing system. Furthermore, while the application program 154 isillustrated in a single data processing system, as will be appreciatedby those of skill in the art, such functionality may be distributedacross one or more data processing systems in, for example, the type ofclient/server arrangement described above. Thus, the present inventionshould not be construed as limited to the configurations illustrated inFIG. 6A but may be provided by other arrangements and/or divisions offunctions between data processing systems. For example, although FIG. 6Ais illustrated as having various circuits and modules, one or more ofthese circuits or modules may be combined or separated without departingfrom the scope of the present invention.

FIGS. 6B and 6C illustrate that the sensitivity lens 100 can beassociated with an icon (shown as a question mark) that can be a “clickand drag” tool on a tool menu or tool bar that a user can drag tooverlie a region of interest in a displayed image 100 i. The sensitivitylens tool 100 can be selectable from a drop down menu, and/or may beactivated via a voice prompt or other user input tool such as upon aleft or right click of a mouse. The sensitivity lens tool 100 may bearranged to have pre-defined alternate visualization and processingparameters to generate the sequence of images based on a similarexamination type. The sensitivity lens 100 may be automatically appliedbased on the examination type without requiring a user to apply the lensto an important diagnostic feature. The sensitivity lens 100 can beconfigured to have a user adjustable size and/or shape to fit the regionof interest with important diagnostic features.

As shown in FIG. 7, uncertainty in a rendered image 100 i from avisualization system 10 can come from sources other than alternativevisualization parameters 101, including, for example, compression 102,post-processing 103, reconstruction 104, and data capture 105. The datacapture 105 is a measurement that typically includes noise. The processof reconstructing 104 the data capture measured values (e.g.,reformatting into a Cartesian grid) can introduce some distortion.Post-processing 103 such as edge enhancement or noise reductionintroduces another deviation from the original data. A lossy compressionalgorithm or other compression technique 102 may also cause distortionin some form. It may be “ideal” to consider all the sources ofuncertainty when exploring alternative appearances (visualizations) ofthe image. In practice, however, it is likely that uncertaintyinformation is known for only some of the components, e.g., the latterthree components in the pipeline: post-processing 103, compression 102,and visualization 101.

In some embodiments, visualization uncertainty can be associated withgreyscale windowing. For example, as shown in FIG. 8, the numericalvalues of the data can be mapped to a greyscale value on screen. Thegreyscale window can be the main tool for controlling the appearancewhen displaying 2D (greyscale) images. In one form of a user-controlledtool, the window is defined by its width and center, corresponding tocontrast and brightness, respectively. The user can interact with thegreyscale window during a diagnostic review, to see if alternativeappearances can lead to other conclusions. The various possiblegreyscale window settings can constitute an uncertainty in thevisualization of the displayed image.

In other embodiments, visualization uncertainty can be found in 3Dimages created by Direct Volume Rendering (DVR) (typically provided asan unclassified data set). In this case, a Transfer Function (TF) mapsnumerical values to optical properties: color and opacity. As discussedabove with respect to the greyscale windowing, adjustments to the TF arerelatively common and can constitute an image uncertainty. Indeed, someradiologists have used the TF as an interactive tool for fuzzyclassification of tissues. FIGS. 9A and 9B are graphs of transferfunction models with material-specific components. FIG. 9A is a graph ofopacity versus intensity of a conventional model where each materialcorresponds to a varying opacity across the data value range(intensity). FIG. 9B is a graph of a likelihood (probability) versusintensity of a probabilistic model, where each material corresponds to avarying material likelihood across the data value range (intensity) andboth a static color and a static opacity according to embodiments of thepresent invention. Thus, the TF can also be interpreted as beingexplicitly probabilistic as shown in FIG. 9B. In this case, thenumerical values are mapped to a set of material probabilities. Theseprobabilities are a direct representation of the possible appearances,i.e., the image uncertainty. In some embodiments, where used, the TF canperform a classifying role that can be separated from itsappearance-mapping role. Each frame of an animated sequence of imagescan be a DVR image showing visualization differences that a user canreadily comprehend and appreciate and is a relatively fast method ofviewing alternative renderings without decreasing spatial precision inthe image.

Generally stated, the two primary functions or roles of TFs can bedistinguished. The TF is seen as a classification tool in both roles,but there is a difference as to whether material probabilities are animplicit or explicit part of the mapping. In the explicit case, theapplication of a TF is typically modeled as a two-step approach. First,the sample value s is mapped to a set of material probabilitiesp_(m)(s), where m is the index among the M materials. Then the materialprobabilities are used to combine the individual material colorsc_(m)=(r_(m), g_(m), b_(m), α_(m))^(T), which results in the samplecolor c(s). Such an approach was employed in the initial DVRimplementation by Drebin et al., according to equations 1 and 2. See,e.g., Drebin et al., Volume rendering, In Proceedings of ComputerGraphics and Interactive Techniques, volume 22, pages 65-74, ACMSIGGRAPH, 1988. Transparent regions are referred to as the null material(m=0).

$\begin{matrix}{{c(s)} = {\sum\limits_{m = 0}^{M}{{p_{m}(s)} \cdot {\overset{\sim}{c}}_{m}}}} & (1) \\{{\overset{\sim}{c}}_{m} = \left( {{\alpha_{m}r_{m}},{\alpha_{m}g_{m}},{\alpha_{m}b_{m}},\alpha_{m}} \right)^{T}} & (2)\end{matrix}$

In the implicitly probabilistic view, the TF is seen as a direct mappingfrom sample value s to sample color c(s). This is the currentlydominating approach and it is the view represented in recent DVRliterature. See, C. D. Hansen and C. R. Johnson, editors. TheVisualization Handbook. Elsevier Butterworth-Heinemann, 2005; and K.Engel, M. Hadwiger, J. Kniss, C. Rezk-Salama, and D. Weiskopf. Real-TimeVolume Graphics. A. K. Peters, Ltd, 2006. Fuzzy classification istypically achieved by connecting material probability to the opacitylevel, giving low opacity to uncertain materials. In the terms ofEquation 1, the product p_(m)(s) α_(m) is integrated into α′_(m)(s).

One objective of the uncertainty visualization contemplated byembodiments of the instant invention is to explore relevant alternativerenderings, given a TF. The implicit representation may not be suitablefor this task. A crude form of sensitivity analysis can be made byperturbing the TF parameters, but control of the exploration may not besufficient. In contrast, an explicit probability model can provide awell-defined uncertainty domain.

Thus, in some embodiments, the TF model used to assess uncertainty canbe explicitly probabilistic. Each material to be visualized is connectedto an individual TF component having two parts, as shown in FIG. 10A.The first is the material appearance (rgbα), which can be chosen to bestatic in order to promote simplicity in the interpretation of therendered image. The second part is the classifying function {tilde over(p)}_(m)(s) that maps intensity to material likelihood. In theexplicitly probabilistic TF model, the familiar TF GUI components definethe material likelihood. FIG. 10A illustrates an exemplary explicitlyprobabilistic TF model, with the TF GUI components defining the materiallikelihood and different colors versus intensity (c₁, c₂ and c₃ allbeing different colors). In this embodiment, the material appearance isseparated from the classification process. FIG. 10B illustrates theexplicitly probabilistic TF model used to generate the animation framesshown in FIGS. 4A-4C. The material appearance is separated from theclassification process. The intensity-specific material probabilitiescan be the normalized likelihoods as expressed in Equation (3).

$\begin{matrix}{{p_{m}(s)} = \frac{{\overset{\sim}{p}}_{m}(s)}{\sum\limits_{m^{\prime} = 0}^{M}{{\overset{\sim}{p}}_{m^{\prime}}(s)}}} & (3)\end{matrix}$

The user can define {tilde over (p)}_(m) (s) in a TF GUI, a convenientform is trapezoids in the [0.0,1.0] likelihood range. Null materiallikelihood can be implicitly defined based on Equation 4.

$\begin{matrix}{{{\overset{\sim}{p}}_{0}(s)} = {\max\left( {0.0,{1.0 - {\sum\limits_{m = 1}^{M}{{\overset{\sim}{p}}_{m}(s)}}}} \right)}} & (4)\end{matrix}$

A standard TF having material-specific components can be transformedinto a probabilistic formulation. The proposed mapping may not giveuseful results in all cases, but it is feasible for certain TF types.The material appearance c_(m) is set to (r_(m), g_(m), b_(m),{circumflex over (α)}_(m))^(T), where {circumflex over (α)}_(m) is themaximum opacity of the material. The material likelihood can be a simplescaling of the opacity profile per Equation (5).{tilde over (p)} _(m)(s)=α′_(m)(s)/{circumflex over (α)}_(m)  (5)

The benefits of explicitly probabilistic TF models can be used foruncertainty animation as will be described further below and in otherapplications. Having a separate probability component can allow manypossibilities for visualization of statistical properties and may alsobe used for non-animated renderings. An example of the use of theexplicitly probabilistic TF model is to connect uncertainty to colordesaturation, which has previously been proposed in other domains. See,e.g., T. Hengl, Visualization of uncertainty using the HSI color model:computations with colors, In Proceedings of the 7th InternationalConference on GeoComputation, pages 8-17, 2003. According to someembodiments of the present invention, with explicit probabilities, it isrelatively straightforward to achieve this effect as shown, for example,in FIGS. 11A and 11B.

FIGS. 11A and 11B illustrate modeling probability by desaturation for anMR renal angiography. FIG. 11A is a traditional rendering, where it canbe hard to visually distinguish between thin tissue regions and regionswith low tissue classification probability. FIG. 11B is an imagedesaturating uncertain regions using a probabilistic DVR approach. TheTF has a single component describing the likelihood of vessel tissue.

It is contemplated that using the TF model of a material-likelihood GUIcan promote the physicians' understanding of the TF adjustment processas an exploration in form of classification interaction rather than anappearance modification.

With respect to uncertainty due to compression, lossy compressionalgorithms will discard information from the data set. Nevertheless,lossy compression is feasible and necessary in many situations withlimited system resources such as network bandwidth or storage capacity.Images displayed from a lossy compressed data set therefore constitute a“best guess”, with inherent uncertainty arising from the unknowndiscarded data. If the compression-related uncertainty can be modeledfor the different regions of the data set, it can be employed in asensitivity lens application. A possible solution for modelingcompression uncertainty is to retrieve distortion information atcompression time and simplify this information into compact meta-datathat can be used to approximate uncertainty in a sensitivity lensapplication. Such a simplification may be achieved by deriving averagedistortion for a number of intervals in the spatial frequency domain orderiving variance for different spatial regions.

Post-processing algorithms, such as edge enhancement filters, high orlow pass filters, anisotropic diffusion and/or sub-sampling filters,combinations of filters or other filters, can alter the data. Typically,there are many possible parameter values for the algorithms that resultin slightly different output images, which can potentially lead todifferent diagnostic conclusions. Normally, a single choice of parameteris made (and may be pre-defined by a technician that is not theradiologist) resulting in a single image instance. The sensitivity lenscan be used to show the image uncertainty arising from multiplepost-processing algorithm parameter settings. The sensitivity lens canalso be used to alternate between applying no post-processing andseveral different post-processing algorithms.

Whether for visualization or other processing parameters that can resultin image uncertainty, the sensitivity lens 100 can be used to generateanimation of a region of interest to show alternative appearances. Insome embodiments, the different uncertainty types can be connected todifferent animation techniques. Many uncertainty types can arise fromdifferent parameter settings. For the user-controlled parameter settingin the greyscale window and TF cases, a useful animation is to performautomatic perturbations (variations) of one or more selected parameters.For example, a base setting is used, then the sensitivity lens 100 isapplied and shows the different appearances due to perturbations of thebase setting parameter(s). The base setting(s) can be predefined or setmanually.

With respect to post-processing operations, post-processing algorithmsare typically performed in isolation from the visualization scheme.Therefore, the animation does not control the post-processing parametersand the above perturbation techniques is not suitable. In this case, theanimation can, instead, rely on meta data describing the alternativeversions of the data. The straight-forward possibility is to have thefull alternative versions, making the animation a sequence steppingthrough the alternatives. As this solution may be unduly memoryconsuming, more efficient representations of the meta data could beused, for instance a low-resolution variance measure for the data set.The sensitivity lens can generate a sequence of images (frames) with ananimation scheme employing a variance-controlled distortion whensampling the data set, regions with low variance (low uncertainty) willyield similar values across the animation frames, whereas high variance(high uncertainty) regions will change over time. Thus, rendering can beanimated by sampling the probability domain over time, which results indisplay of a varying appearance for uncertain regions.

A specific case of uncertainty arising in a visualization stage is whenemploying DVR with a probabilistically interpreted TF. Even though fewTFs are described as explicitly probabilistic, many models can beconnected to such an interpretation. The animation approach can be usedto represent material probability in the time dimension as will bediscussed further below. In some embodiments, the animation can providethe arrangement of material mappings across time, resulting in aplurality (e.g., three) of different animation modes.

In some embodiments, animation techniques for exploring the domain ofpossible outcomes can use a probabilistic classification. Generallystated, the approach translates to a sequence of Monte Carlo simulationsof the rendering. The classifier in focus can be the probabilistic TFmodel discussed above that can provide a set of material probabilityvalues {p_(m)(s)}, m=0, 1, . . . , M, for each intensity value “s”. Theanimation techniques can, however, be employed in combination witharbitrary probabilistic classification method.

The animated rendering can be the derivation of the sample color c(s)from the material probabilities. In a traditional rendering, the processwould be to mix the materials' colors, weighted by the probabilities asdescribed by Equation 1. Introducing animation, the possibility torepresent probability in the time dimension is added. A straight-forwardlinear mapping is employed. Having p_(m)(s)=x % translates to settingc(s)=c_(m) in x % of the frames in the animation. Doing this for allmaterials, the sequence of sample colors captures the probabilisticproperties of the sample. Applying this for all samples in the volumewill, consequently, capture the probabilistic properties of the entiredata set, see FIGS. 14 a-k. Intensity values corresponding to uncertainmaterial classification will have varying color mapping in therendering. Note that to achieve a relevant probabilistic representationin the animation, the material colors cannot be mixed in the samplingstage. Color mixtures in the rendering only arise from depth-wisecompositing. Those of skill in the art will recognize thattime-probability representation is not the only possibility forrendering.

Θ denotes the total number of steps in the animation cycle. Thus, theprobabilistic animation scheme can map each intensity value to an arrayof Θ material colors, in contrast to the single rgbα value in thetraditional case. This is illustrated in FIG. 12. Expanding this to thewhole intensity range, the normal 1D look-up of material color mixturesis replaced by a 2D look-up of pure material colors. The added dimensionis named the animation index, θ. The animation is achieved by cyclicallychanging the θ value for each rendered frame. The resolution in theprobability domain is controlled by increasing the Θ value. Therenderings of medical data sets in some examples in this applicationhave Θ=16.

FIG. 12 schematically shows a traditional DVR vs. probabilisticanimation. Traditionally, the TF maps an intensity value to a singlecolor, a mixture of material colors as noted by a). The probabilisticanimation maps an intensity value to an array of pure material colorswith an animation index θ as noted by b).

The animation has been implemented as part of a GPU-basedtexture-slicing volume renderer. Typically, the only modification of thestandard implementation is to replace the color-opacity look-up table(LUT) by a material selection LUT and a material color array, see FIG.13.

The material selection LUT is a 2D table, where each row spans theintensity range and the columns correspond to the Θ steps in theanimation cycle. The table is filled with material indices. Thus, eachrow represents a specific intensity

material mapping for an animation frame. The found material index pointsto the appropriate position of the material color array, which yieldsthe fragment color.

FIG. 13 illustrates a fragment shader implementation of probabilisticuncertainty animation. From left to right, the sample intensity value sis a column index into the material selection LUT. Together with theanimation index θ, a material index is looked up and finally used tofind the resulting sample color.

FIGS. 14 a-k illustrate examples of traditional and probabilisticanimation. Although shown in black and white, the images are typicallyin color and as shown were rendered in red (the lighter gray portion)and blue (the darker/black portion). The data set has a linear gradientin the horizontal direction. FIG. 14 a is a traditional rendering. FIG.14 b is a corresponding traditional TF used for the rendering in FIG. 14a. FIG. 14 c is the explicitly probabilistic TF used for the animations.FIGS. 14 d-f are frames from probabilistic animation (Θ=4) in randommode, for θ=0, 1, and 2. FIG. 14 g is a schematic view of intensity

color mapping in random mode. FIGS. 14 h-j are frames from probabilisticanimation (Θ=4) in material sync mode, for 0=0, 1, and 2. FIG. 14 k is aschematic view of intensity

color mapping in material sync mode.

The material selection LUT can be derived in software as soon as the TFis changed. For each frame to render, the animation index θ is updated,which is used to sample the correct row of the material selection LUT.The discussed methods affect all samples of a certain intensity equally.However, an alternative would be to perform an individual material colorlookup for each sample, i.e., changing θ between samples. This, however,may have certain disadvantages in terms of computational complexity andreduced difference between animation frames.

The above-discussion focuses on the simple classification of a 1Dintensity TF. The general case is that an arbitrary classificationscheme has produced a volume with material probabilities. Theprobabilistic animation is expected to generalize well as will bediscussed further below.

In some embodiments, different animation modes may be used, depending,for example, on the degree of variation or the type of uncertainty beingexplored. As described above, the material probabilities connected to acertain intensity value can be represented by the relative number ofentries of each material in the corresponding column of the materialselection LUT. The term “animation array” will be used for such acolumn. The probabilistic relevance is retained as long as the number ofentries for each material is not changed, but the order of the entriescan be changed without loosing the probability mapping. Differentuncertainty visualization effects can be achieved by arrangement withinthe array. For example, a plurality of different animation modes can beused. FIG. 15 illustrates three probabilistic animation modes.

Random mode. The population of the array is fully randomized.

Grouped random mode. The entries are grouped for each material, but theorder and position of the groups are randomized.

Material sync mode. Each material has a fixed base position in thearray, around which the material entries are centered.

FIG. 15 shows that the respective modes are defined by the arrangementof material entries across the animation index dimension, illustratedfor a single intensity value. Note that the number of entries for eachmaterial is the same across all modes. (Black=null material) The leftside of the figure illustrates the Random mode. The materials arerandomly inserted according to the probabilities. The middle gridrepresents the Grouped random mode. The entries are grouped according tothe material and the groups randomly placed. The right grid illustratesthe Material sync mode. The materials are centered at a respective baseposition.

One motivation for the grouped random mode is that it has a smootherappearance relative to the random mode, which mitigates the visualfatigue of looking at a flickering image. The benefit of the materialsync mode is that there will be an animation frame that shows themaximum extent of each material, at the animation index corresponding tothe material's base position. This synchronization uses a sortingprocedure for each value in the sample range and can be performed asfollows. The algorithm can produce an array containing Θ materialselections for each intensity value. A common base allocation span d andmaterial base positions μ_(m) are set:

$\begin{matrix}{{d = \left\lfloor {\Theta/\left( {M + 1} \right)} \right\rfloor},{u_{m} = {{m \cdot d} + \left\lfloor \frac{d}{2} \right\rfloor}},} & (6)\end{matrix}$A selected material is then entered into the array at the positionclosest to the base position. The base allocation span is reserved forthe corresponding material for its first d selections, if there are somany. When all possible base allocations are done, the remaining emptyslots are filled with materials having a number of entries exceeding thebase allocation.

Variations of the animation techniques can be put to use in differentscenarios. An immediate, qualitative impression of the uncertainty isachieved by the random or grouped random modes at a fairly highanimation speed (e.g., 10 fps or “frames per second”). For more detailedanalysis of possible extents of a tissue, the material sync mode at amoderate animation speed (e.g., 5 fps) is appropriate. Anotherpossibility is to use the material sync mode for TF tuning. Using a lowanimation speed (e.g., 2 fps), the user can fix the TF at the animationstep that best represents the features of interest.

In some embodiments, the instant invention can be used in workflowsettings. That is, in some embodiments, the sensitivity lens 100 can beused when the radiologist gets a predefined setting of some sort andwants to check that the appearance in diagnostically important regionsof the image is robust across alternative settings, without resorting totime-consuming and uncontrolled manual adjustment.

In the clinical case, the sensitivity lens 100 can be configured to meetextreme demands on simplicity and efficiency since the physicianstypically have very limited time for each diagnosis. In someembodiments, uncertainty animation can be used on a small region ofinterest rather than an entire volume. The typical case is that asensitivity analysis is wanted for a few clinically important regions.The lens 100 can be a small user-controlled region-of-interest in whichthe selected animation technique is applied while the rest of the imagecan be rendered traditionally. This tool is expected to be easy tomaster, since it can be used without parameter tuning and since itresembles a familiar magnifying glass tool. Another advantage is thatpotential visual fatigue from studying the animations is reduced thanksto the small region.

The sensitivity lens 100 has potential to become an important tool inthe clinical workflow for 3D visualization with DVR. As a consequence ofthe high radiologist workload, there are many radiologists that do nothave the time to become 3D experts. In those cases, a common solution isthat a technician works as the 3D expert and produces the base settingsfor the 3D visualization, including the TF parameters. The radiologistthen makes the diagnostic assessment from this predefined visualization.Unfortunately, this workflow model may introduce risk since the limitedmedical knowledge of the technician may lead to inadequatevisualizations without the radiologist noticing it. The sensitivity lens100 provides an efficient tool to evaluate the robustness of thetechnician's settings while not requiring the radiologist to know theintricate details of DVR. The sensitivity lens 100 can then act as aquality assurance tool, allowing the radiologist to explore alternativerenderings substantially automatically.

As noted above, the probabilistic animation scheme has been primarilydescribed using a classification in form of a 1D intensity-based TF. Therelevant starting point for an arbitrary classification scheme is avolume with material probabilities. However, embodiments of theinvention have broader application. For example, as shown in FIG. 16,the probabilistic animation can be used for the general case.

FIG. 16 illustrates a volume classified with respect to M materials,exemplified for M=3. From left to right: The material probability vectorp=(p1, . . . , p_(M)) is looked up in the material probability LUT,restricted by Σ^(M) _(m=1)p_(m)≦1.0. The found value is a column indexinto the Θ×N material selection LUT. Together with the animation indexθ, a material index can be looked up and used to find the resultingsample color.

A sample from the classified volume corresponds to a location in anM-dimensional material probability LUT. The material probability LUTpoints into a material selection LUT and the sample color is then foundin the material color array.

The sizes of the LUTs are not expected to be a problem. Judging from theexperience with the TF-based scheme, a size of 10^(M) would besufficient for the material probability LUT and can probably be reducedfurther for large M. Moreover, the restriction that the sum ofprobabilities is less than 1.0 allows for reduced storage. The number ofcolumns in the material selection LUT, N, equals the number of uniqueways to fill Θ slots from a set of M+1 material indices. This is knownas the multichoose operation and can be mathematically described asfollows below.

$\begin{matrix}{{{N\left( {M,\Theta} \right)} = {\begin{pmatrix}{\left( {M + 1} \right) + \Theta - 1} \\\Theta\end{pmatrix} = \frac{\left( {M + \Theta} \right)!}{{M!}{\Theta!}}}}\begin{matrix}{{N\left( {4,10} \right)} = 286} & {{N\left( {8,10} \right)} = 43758} & {{N\left( {4,20} \right)} = 63756}\end{matrix}} & (7)\end{matrix}$Thus, the LUT size is not a problem for reasonable values of M and Θ.

The present invention is explained further in the following non-limitingExamples Section.

EXAMPLES

The tests of the proposed animation techniques evaluated aspects of theclinical usefulness. The results are presented below in two parts.First, results from an experimental study on stenosis assessment forsimulated vessels are shown. The second part includes renderings of datasets from diagnostic situations where the classification task of DVR isthought to be particularly challenging and sensitive to errors. Notethat the still image presentation in this paper makes the frame-to-framedifferences appear overly subtle. Refer to the video file available athttp://www.cmiv.liu.se/Members/clalu/uncert.mpg for a more realisticimpression of the animation.

Simulated Clinical Task

A test of the benefits of uncertainty animation is to find out whetherit can increase the diagnostic accuracy in the challenging case of MRangiographies. Therefore, an experimental study on simulated vessels wascarried out, see FIGS. 17A-17D. The test subjects were twelvephysicians, eleven radiologists and one cardiologist, all with clinicalexperience of stenosis assessment.

FIGS. 17A-17D illustrate a simulated stenosis assessment. The outerperimeter of the vessel model has constant radius, whereas there is aconcentric plaque that creates a simulated stenosis. The plaque profileis a cosine curve. The task is to assess the stenosis degree, d1/d2 asshown in the schematic of FIG. 17D. As also shown in FIG. 17D, thevessel model has three materials, the vessel lumen, the vesselwall/plaque, and the background with overlapping intensity distributionsin order to make the task challenging. FIG. 17A is an example of arendered image frame in material sync mode. FIGS. 17B and 17C arecut-outs (corresponding to the red frame of full image 17A) of thevessel at two other time steps. FIG. 17D is a schematic of the vesselmodel.

The test was designed to resemble a low-quality data set in combinationwith an untuned and fixed TF, when zooming in on a suspected stenosis.Three rendering methods were tested for 24 vessels: traditional staticrendering and uncertainty animation in grouped random and material syncmodes. The animations were fixed at 6.0 fps. The accuracy of each methodwas measured as the absolute error in assessing the stenoses on afour-grade scale. The details of the study setup are given below.

The result of the study is presented in Table 1, method results to theleft and pairwise comparisons to the right. The scores are aggregatedassessment error, a lower score means a higher accuracy. The scorespread is presented as interquartile range (IQR). The statisticalsignificance level was set at p<0.05 and the statistical analysis wasperformed in StatView 5.0 (SAS Institute, Cary N.C., USA).

TABLE 1 Error Scores and Comparisons for Stenosis Assessment MethodMedian IQR Material Sync 7.5 1.0 Grouped Random 13 3.5 Traditional 147.5 Comp. p Material Sync < Traditional 0.0057 Material Sync < GroupedRandom 0.013 Grouped Random < Traditional 0.35

The result shows that the material sync mode provides the highestaccuracy in this task. The aggregated error for each subject/methodcombination was analyzed using Friedman's test, which showed that thereis significant differences between the methods (p=0.0069). Subsequentpairwise Wilcoxon tests showed that the material sync mode hassignificantly lower error (higher accuracy), than the other two methods.The grouped random animation has slightly lower median than thetraditional rendering, but the difference is not significant.

In order to establish a “gold standard” benchmark, the subjects alsoassessed the 24 vessels in a traditional rendering with free manualadjustment of the TF. The median accuracy was found to be 6, which meansthat the material sync mode, with a median of 7.5, comes quite close tothe gold standard even though no TF adjustment was allowed in that case.

A brief interview was carried out with the physicians after the test.Out of the tested methods, ten out of the twelve preferred the materialsync mode in the test setup. Two reasons repeatedly came up asmotivation: that the vessel structure was clearly seen and that thesynchronization resembled the familiar manual back-and-forth TFadjustment. Two physicians preferred the grouped random mode, but theiraccuracy was actually lowest for that method. Furthermore, there wasconsensus that uncertainty animation would improve accuracy and/orefficiency for difficult stenosis assessments. Finally, nearly allphysicians stressed that in real clinical usage it is highly desirablethat the physician be able to interactively control the uncertaintyanimation in terms of the speed and the underlying probabilistic TF.

Details of Experimental Study

The vessel model, shown in FIGS. 17B-17D, was validated as realistic bythree radiologists before the start of the study. In order to make theassessment challenging, the three materials (vessel lumen, vesselwall/plaque and background) were given highly overlapping intensitydistributions. A realistic appearance of the vessel lumen was achievedby making the intensity decrease for increasing radius, causing thetransition to the lower intensities of the wall and plaque to becomevery subtle. Twenty four (24) different vessels were tested withrandomized variations in stenosis degree, but also other parameters(radius and horizontal position of vessel, vertically varying horizontaloffset of vessel, and vertical position of stenosis) were given varyingvalues to make each test independent. Each vessel was given a randomizedTF, consisting of a single trapezoid, designed to be more or lesssuboptimal. The transformation of Equation 5 was used to createcomparable TFs for the methods. The TFs and the viewpoint were notadjustable.

Each physician assessed the stenosis of the vessels on a four-degreescale: 0-25%, 25-50%, 50-75%, and 75-100%. For the case that thestenosis was not assessable at all, the radiologist had a fifth possibleoutcome, “Unknown”. The scale was indexed 0, 1, 2, 3 and the absoluteindex difference to the target index was used to measure accuracy. The“Unknown” response was given a fix error of 3.

In total, each test consisted of 72 trials, i.e., all method/vesselcombinations. The order of the trials was determined by a controlledrandomization. To avoid carry-over effects, the order in which themethods occurred was counterbalanced both within each subject's trialset and between the subjects. Moreover, the controlled randomizationensured that two trials for a certain vessel was separated by at least20 other trials. To familiarize the subjects with the task before thetest, the setup was demonstrated by 12 dry-run trials (allmethod/stenosis degree combinations) where the TF could be adjusted andthe accuracy of the assessment was fed back to the subject. The “goldstandard” test with free manual TF adjustments was carried out for the24 vessels after the main test.

Renderings of Clinical Data Sets

MR angiographies are a clinical examination with great need foruncertainty visualization. MR examinations are preferred over CT scanssince ionizing radiation is potentially damaging and is to be avoided ifpossible. DVR of MR angiographies is, however, time-consuming anderror-prone. Static TFs are not used as the intensity scale variesbetween patients and an inadequate visualization can give an incorrectimpression of the vessels that affects the medical assessment. See,e.g., Persson et al., Volume rendering compared with maximum intensityprojection for magnetic resonance angiography measurements of theabdominal aorta. Acta Radiologica, 45:453-459, 2004. The animatedrenderings have been applied to several MR angiographies, two examplesare described below.

FIGS. 18A-18C show the uncertainty animation in material sync mode for aMR renal angiography. The diagnostic task is to determine location andsignificance of any stenoses. The vessels cannot be isolated through anysingle setting of an intensity-based TF; when the minor vessels appear,so do the kidneys and other parts. With the uncertainty animation, thealternative visualizations can be explored by a single TF setting. FIG.18D is screen shot from the test application GUI and FIG. 18E is aprobabilistic TF.

Despite the calibrated Hounsfield scale, CT data sets often haveoverlapping tissue intensity ranges, which can cause ambiguousvisualizations. Such an example is the thyroid tumor in FIGS. 19A-19F,where the tumor extent in relation to the left carotid arteries iscrucial to the pre-operative planning. The highly different frames showthat there may be a large span of alternative visualizations even thoughthe traditional, static rendering seems distinct.

FIGS. 19A-19D are images of a CT examination of a thyroid tumor. The topimage (FIG. 19A) is a traditional diagnostic visualization withoverlapping TF components for tumor (green), vessels (orange), and bone(white). The following three images (FIGS. 19B-D) are different framesfrom an uncertainty animation in material sync mode. The dramaticdifferences between the frames show that there is much inherentuncertainty in the traditional rendering (FIG. 19A) to be explored. Thetraditional TF is shown in FIG. 19E and the probabilistic TF is shown atFIG. 19F.

Another example is an MR brain examination showing a large cyst, asshown in FIGS. 20A-20D. FIG. 20A illustrates a tissue complex in theright brain hemisphere having a cyst (a), a hemorrhage (b), and aventricle (c), all three in the yellow-orange span. The complex issurrounded by an edema (green). The diagnostic task is to determine theborders between the cyst and the hemorrhage. FIG. 20A is a traditionalrendering. FIG. 20B is a probabilistic TF. FIGS. 20C and 20D are twodifferent zoom-ins of uncertainty animation frames in random mode.

The important clinical question is if the cyst is distinctly separatedfrom the hemorrhage, the two regions having very similar intensities.The uncertainty animation provides a controlled exploration of theextent of the tissues.

On the tested system, the rendering performance was about 10% lower forthe animation methods compared to the standard method, see Table 2. Thederivation of the material selection LUT was carried out in about 1-3ms. The test system was a PC with an AMD Athlon 64 CPU and an nVidia8800 GTX graphics board with 768 MB, the volumes were rendered in a512×512 view port.

TABLE 2 FPS Performance of Uncertainty Animation Data set MR renal CTneck MR brain Size 512 × 512 × 56 512 × 512 × 512 432 × 432 × 160Traditional 42.7 9.2 31.0 Animation 38.4 8.2 28.4

The experimental study shows that for a fixed TF setting, theuncertainty animation in material sync mode is clearly more efficientfor stenosis assessment than a static rendering. In fact, it comes quiteclose to the “gold standard” of free manual adjustment, a clearindication that time-consuming TF interaction can to some extent bereplaced by uncertainty animation. Even though the intentionally poorconditions of the simulation would not be acceptable in a clinicalsituation, the physicians considered the test to have bearing on realdiagnostic work.

The foregoing is illustrative of the present invention and is not to beconstrued as limiting thereof. Although a few exemplary embodiments ofthis invention have been described, those skilled in the art willreadily appreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention as defined inthe claims. The invention is defined by the following claims, withequivalents of the claims to be included therein.

That which is claimed:
 1. A system for rendering images from amulti-dimensional data set, comprising: a visualization system with adisplay and a graphic user input configured to allow a user toselectively apply a sensitivity lens to a region of interest of aportion of an image shown on the display, the image rendered from amulti-dimensional image data set, wherein the sensitivity lens causesthe visualization system to automatically electronically render at leastone different visualization of the portion of the image associated withthe sensitivity lens to interrogate uncertainty of at least one featurein the region of interest of the portion of the rendered image.
 2. Asystem according to claim 1, wherein the multi-dimensional data set is amedical image data set, and wherein the at least one differentvisualization is a sequence of different rendered visualizations of onlythe portion of the image associated with the sensitivity lens.
 3. Asystem according to claim 1, wherein, the sensitivity lens is a windowthat can be electronically moved and placed by a user over a selectedportion of the image on the display, and wherein, in response to a userselectively applying the sensitivity lens, the visualization systemdisplays a series of different rendered visualizations of the portion ofthe image underlying the sensitivity lens to show to a user uncertaintyof at least one feature in the portion of the image under thesensitivity lens.
 4. A system according to claim 1, wherein the at leastone different visualization is a plurality of different renderedvisualizations that illustrate only the portion of the image associatedwith the sensitivity lens in different manners using at least one of thefollowing: (a) different viewing parameters; or (b) different processingparameters, to thereby allow a clinician to consider uncertainty in theportion of the rendered image associated with the sensitivity lens.
 5. Asystem according to claim 1, wherein the sensitivity lens is a windowwith a user adjustable size, a user adjustable shape, or a useradjustable size and shape that can be moved and placed over a selectedregion of interest, and wherein the sensitivity lens causes the displayto show a series of different views of the underlying portion of theimage to allow a user to visually assess uncertainty in the renderedvisualization of the underlying region of interest.
 6. A systemaccording to claim 3, wherein the sensitivity lens is configured to zoomthe region of interest to thereby enlarge features in the region ofinterest in the different visualizations while the remaining imageoutside the sensitivity lens remains substantially unchanged.
 7. Asystem according to claim 1, wherein the sensitivity lens is configuredto generate a sequence of different visualizations of the portion of theimage using perturbation of viewing parameters.
 8. A system according toclaim 7, wherein the visualization system is configured to generate atwo-dimensional greyscale image of the portion of the image associatedwith the sensitivity lens.
 9. A system according to claim 8, wherein thesensitivity lens is configured to use alternate greyscale windowingparameters to generate alternate visualizations of the two-dimensionalgreyscale image thereby assessing greyscale windowing uncertainty in therendered images.
 10. A system according to claim 1, wherein themulti-dimensional data set is a volumetric or time varying volumetricdata set, and wherein the visualization system is configured to rendervisualizations with Direct Volume Rendering.
 11. A system according toclaim 10, wherein the sensitivity lens is configured to generate asequence of different visualizations of only the portion of the imageassociated with the sensitivity lens using different attributes ofTransfer Functions.
 12. A system according to claim 11, wherein thesensitivity lens displays the sequence of the different visualizationsof the portion of the image so that tissue associated with at least onefeature in the region of interest is visualized with differences in atleast one of opacity, color luminance and color saturation.
 13. A systemaccording to claim 2, wherein the sensitivity lens is configured todisplay the sequence of different visualizations of the portion of theimage so that at least one of different visualizations is rendered usingimage data discarded during compression.
 14. A system according to claim2, wherein the sensitivity lens is configured to display the sequence ofdifferent visualizations of the portion of the image so that at leastone of the different visualizations is rendered using either: (i)visualization data; (ii) image data; or (iii) visualization and imagedata not used to render one or more of the other differentvisualizations.
 15. A system according to claim 2, wherein at least someof the images in the sequence of are generated using probabilisticanimation based on a probabilistic transfer function model to classifymaterials in the image data set.
 16. A system according to claim 1,wherein the system comprises a medical visualization pipeline with atleast one server and at least one client.
 17. A system according toclaim 2, wherein at least some of the frames in the sequence ofvisualizations are configured to display uncertain regions with avarying appearance from that of other frames.
 18. A system according toclaim 2, wherein the sequence is configured to display in differentanimation modes depending on a type of uncertainty in the visualization.19. A system according to claim 2, wherein the sequence is configured todisplay in a material sync mode.
 20. A system according to claim 1,wherein the sensitivity lens causes the visualization system toautomatically electronically render different visualizations of only theportion of the image with the region of interest and display thedifferent visualizations in at least one of material sync animation,random mode animation and grouped random mode animation.